Mental Images of Text: Learning Document Similarity using Web Photos

Modern search engines rely solely on text to analyze the content of Web documents. However, it is well known that humans often incorporate ”mental visualization” in the form of mental images in order to interpret text. Psychological studies have demonstrated that humans are able to create these visual perceptions even in absence of external visual stimuli. Such a physiological behavior in the human brain suggests that incorporating visual information to text analysis in the machines could be beneficial as well. In this paper, we present a method that incorporates the idea of mental images to learn a document similarity metric exploiting both text and visual data. The key idea behind our method is to associate a visual representation to the most salient portions of the original text. We do so by leveraging text-based image search engines: the photos retrieved using these textual queries can be viewed as akin to the mental images used by humans. We then combine these images with the original text representation in order to perform a joint analysis of text and visual data. In this paper we demonstrate this approach on the task of semantically clustering Web search results for a given query. Our experiments indicate that incorporating images leads to increased accuracy rates relative to systems relying on text only. In addition, our method learns a universal document similarity metric which can be successfully generalized to any queries and arbitrary documents, a property that provides an efficient framework to perform predictions at search time. Finally, we show that the concept of mental images can be successfully applied to different text domains, which suggests that our method can be generalized to many different tasks.